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Abstract 

We rely on data from measurements every day; a gas-pump, a speedometer, and a 
supermarket weight scale are just three examples of measurements we use to make 
decisions. We generally accept the data from these measurements as “valid.” One reason 
we can accept the data is the “legal metrology” requirements established and regulated by 
the government in matters of commerce. The measurement data used by NASA, other 
government agencies, and industry can be critical to decisions which affect everything 
from economic viability, to mission success, to the security of the nation. Measurement 
data can even affect life and death decisions. Metrology requirements must adequately 
provide for risks associated with these decisions. To do this, metrology must be integrated 
into all aspects of an industry including research, design, testing, and product acceptance. 

Metrology, the science of measurement, has traditionally focused on the calibration of 
instruments, and although instrument calibration is vital, it is only a part of the process that 
assures quality in measurement data. For example, measurements made in research can 
influence the fundamental premises that establish the design parameters, which then flow 
down to the manufacturing processes, and eventually impact the final product. Because a 
breakdown can occur anywhere within this cycle, measurement quality assurance has to be 
integrated into every part of the life-cycle process starting with the basic research and 
ending with the final product inspection process. 

The purpose of this paper is to discuss the role of metrology in the various phases of a 
product’s life-cycle. For simplicity, the cycle will be divided in four broad phases, with 
discussions centering on metrology within NASA. 

• Research 

• Design 

• Testing 

• Product Assurance 

The target audience for this paper is persons responsible for policy and program 
requirements for industry and government. 

Introduction 

When the international definition of metrology is considered (the science of measurement 
[1]), the importance of metrology comes into sharper focus. In the simplest terms, 
metrology provides the measurement data used to make decisions. The quality of the 
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decision is directly proportional to the quality of the measurement data used. The old 
computer adage, “garbage in, garbage out” also applies to the decision process where 
measurement data is involved. This can be very obvious during the acceptance of products 
when poor measurement processes or incorrect measuring devices are employed. What 
may not be as apparent, though, is the role metrology plays throughout the life-cycle, even 
long before the final acceptance measurement is made. 

In the broadest sense, all activities which have an impact on the quality of a measurement 
fall within the scope of metrology. This includes activities as early as the basic research 
which establishes the known boundaries of science to the attitude of the technician 
performing the final measurement of a manufacturing process. Unfortunately today, for 
many, metrology has become synonymous with the calibration of measuring equipment. 
Although critical to the measurement process, calibration by itself does not provide 
assurance of accurate measurement data. Such assurance can only come from the proper 
and appropriate application of metrology “best practices” within each of the lifecycle 
phases. These “best practices” could be viewed as analogous to the requirements that 
allow modem international commerce to prosper. 

Background 

There has always been some form of international trade since the earliest human history, 
but only within the last two centuries has it been able to grow exponentially. Metrology 
played the pivotal role that allowed for this expansion with the development of two crucial 
areas: a standardized set of measurement units and agreements in measurement 
requirements for trade. 

The International System of Units (SI) is now the global reference for measurement units. 
In countries that have not fully switched to the SI, consumer goods usually display dual 
units. Conversion, though cumbersome, is standardized. This has not always been the 
case. Historically, units of measurement varied between regions, countries, and towns. 
Many times the units even differed between occupations within the same town. 
Measurement units often changed with changes in government or individual leaders. 
Without agreement on a single measurement system, trade was more difficult and even 
international scientific endeavors were hampered. Abuse and fraud in commerce was 
common [2]. These abuses led to a demand for a common measuring system for grain and 
wine to be added as a clause to England’s Magna Carter of 1 21 5 AD [3]. The need was 
universal, but it was not until the late 1 8 th century that a true standardization of measures 
began to emerge. The French definition of the meter and kilogram became the starting 
point for the development of the SI system of today. International unification of 
measurements finally came with the international treaty Metre Convention in 1875 [4] 
which established the Bureau International des Poids Mesures (BIPM) charged with 
maintaining the SI reference standards. 

The second key to international commerce was to apply standardized rules which would 
provide assurance as to the quality of measurements. Legal Metrology became the branch 
of metrology that is concerned with measurements that directly affect consumers. The 
International Organization of Legal Metrology (OIML) develops the regulations for this 
effort with the main concern being the chain of traceability for measurements. These 
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regulations, concerned with consumer protection, are often incorporated into national or 
international law. 

The largest difference between general metrology and legal metrology is scale of precision. 
While scientists in general metrology may be concerned with such areas as the definition 
or uncertainty of SI standards at the BIPM-level, those in legal metrology are concerned 
with the application at the consumer level [2]. It is from this application that general 
metrology becomes transparent whether the consumer is buying 5 kilograms of potatoes or 
a cargo ship load. 

For NASA and others in industry to have the same “transparent” metrology for making 
decisions based on measurement data, metrology “best practices” must be incorporated 
into each phase of the product lifecycle. These “best practices” should be incorporated 
into the culture of the organization, becoming a fundamental core competency for the 
business of that organization. To facilitate metrology best practices, a need exists to 
implement them across an organization in a manner similar to the requirements of Legal 
Metrology. This will provide assurance as to the quality of measurements. 

The following will be discussions of applying metrology into the different phases of the 
lifecycle. NASA’s Systems Engineering Handbook [5] identifies seven different phases 
for a product lifecycle from concept to closeout. For the purposes of this paper, the phases 
will be much broader and the role of metrology in research will also be discussed. In 
order, the discussion will cover metrology within: 

• Research 

• Design 

• Testing 

• Product Assurance 

The scope of each topic to be discussed could and has filled volumes of written material, 
usually independent of each other. The goal of this paper is to present the topics together 
to illustrate the inter-relationship of metrology. To keep the paper as short as possible, 
each topic will be given only the briefest coverage possible. The most emphasis will be 
given to the design phase, which, in the author’s opinion and experience, is the single area 
where metrology can have the largest return on investment. 

Metrology’s Two Fundamental Tenets 

Before its role throughout the life-cycle can be discussed, metrology’s fundamental tenets 
have to be understood. It is the rigorous application of these two tenets that characterizes 
the success of legal metrology. 

• Traceability 

• Uncertainty 

Traceability establishes the link for a given measurement to the national or international 
standard for that unit of measure. As discussed above, this was the largest hurdle to 
international commerce and scientific cooperation. 

Traceability is accomplished through an unbroken series of competent and documented 
calibrations. Documented calibration is important to the quality of a measurement, 
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especially in areas of research, testing and product acceptance. Figure 1, taken from the 
BIPM website, illustrates this “chain of traceability.” 
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Figure 1: BIPM representation of the traceability flow-down for a kilogram [6]. 


Uncertainty is the second fundamental tenet of metrology. Measurement uncertainty is the 
estimate of the errors in a measurement that results from the equipment, the processes, and 
other sources, including the originating international standard, as well as the quality of the 
traceability chain. Every element within the measurement process contributes errors to the 
measurement result. Evaluating the measurement uncertainty provides an estimate of the 
quality of the measurement [7], Not all tasks require the same level of quality, thus 
measurement uncertainty provides the tool to adjust measurement processes according to 
technical as well as business needs. 

For many years, uncertainty estimation had the same problem as units of measure - a lack 
of standardization. The lack of standardization has caused (and still causes) disagreements 
and confusion. After almost two decades of work, an international consensus standard was 
developed. The International Organization for Standardization (ISO) Guide to the 
Expression of Uncertainty in Measurement (GUM) [8] provides a standardized approach to 
estimating uncertainty. ANSI/NCSL Z540.2-1997 (R2007), U.S. Guide to the Expression 
of Uncertainty in Measurement (U.S. Guide) [9], is the U.S. adoption of the ISO GUM. 
Additional guidance on estimating measurement uncertainty can be found in many 
discipline-centered voluntary consensus standards and complimentary documents. BUT, 


- 4 - 









for consistent results within an organization, among organizations, or across national 
economies, it is imperative that the uncertainty analysis is based on the ISO GUM. 

Research Metrology 

In general terms, research is either basic or applied. Acknowledging there are “gray 
zones,” basic research is usually driven by scientific curiosity, while applied research looks 
for commercial applications, or to solve a specific problem. The role of metrology is the 
same for both types of research and is based almost entirely on the two fundamental tenets. 

One of the most important considerations for research results is independent verification. 
Others must be able to reproduce the work. For this to happen, all measurement data must 
be traceable to a consensus standard (within an organization or international body), an 
international standard, or a standard based on fundamental physical laws. If the measuring 
equipment is not calibrated based on these standards, then the data is not traceable and the 
results may not be reproducible. This throws doubt on the credibility of the research. 

Measurement process uncertainty is another area that has a strong impact on the 
reproducibility of research results. Dr. Les Kirkup relates the role of uncertainty in science 
research by stating, “One overlooked (or incorrectly assessed) source of error could 
transform potential Nobel prize winning work into an amusing curiosity of little or no 
consequence” [10]. For the applied research used by NASA, poor results may be far more 
disastrous than missing the Nobel Prize. 

Design Metrology 

Theodore von Karman said, “Scientists study the world as it is; engineers create the world 
that has never been.” This observation characterizes the difference between research and 
design. Where scientific research explains and quantifies the world around us, design 
engineering uses that information to create products. In the process of creating these 
products, designers must turn the expectations of the end-user (or stake-holder) into 
technical requirements which allow the product to be built. NASA’s System Engineering 
Handbook describes the inter-relationship of the processes which take the requirements 
from concept to final product. The Handbook describes many levels in the design process, 
but for the purpose of this discussion, the scope will concentrate on just two types of 
requirements. 

• Functional requirements. These are the requirements that describe how the end-user 
desires the product to perform. These requirements may be qualitative (e.g., “I 
want the item to be aesthetically pleasing”) or quantitative (e.g., “The spacecraft 
will operate in an orbit between 400 and 650 kilometers”). 

• Design requirements. These are the requirements that establish the physical (e.g., 
size, weight, etc.) and operational (e.g., pressure, RPM, etc.) requirements of the 
product. These requirements are quantitative. 

Functional Requirements 

Functional, or performance, requirements are usually validated after prototype or first 
article development is complete. It is for this reason that the metrology of functional 
requirements should be considered in the early planning phase to properly scope the 
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project. Assessing the measurement requirements in the beginning can save many 
problems at product delivery. Although beyond the scope of this paper, it would be very 
prudent to obtain stakeholder approval of the methods to be employed in the measurement 
of qualitative requirements. Aesthetics, for example, can be very subjective. 

The metrology of the quantitative requirements also should be given adequate attention 
early in the design phase. Not only the required accuracy should be assessed, but also the 
method of measurement used in the validation process. The nature of the functional 
requirements dictates the type of measurement and sometimes existing measurement 
technology is simply not adequate. This can have a very serious impact on the project’s 
schedule and cost. 

A brief description of a current project at NASA’s Jet Propulsion Laboratory (JPL) 
illustrates metrology’s impact on meeting functional requirements. This JPL project is a 
good example of where the measurement capability of satisfying the functional 
requirements can be made a part of the project’s scope. 

The objective of the Small Particle Hyper-Velocity Impact Range (SPHIR) is to 
provide a low-cost hyper-velocity accelerator with velocities up to 10 kilometers 
per second (over 22000 miles per hour). Such a device is needed to develop better 
shielding for manned and unmanned spacecraft. Currently, it is very expensive to 
operate hyper-velocity accelerators in the velocity range required. It becomes cost- 
prohibitive to gather large amounts of test data on hyper-velocity impacts with 
different shielding techniques. Because current technology does not exist to 
measure small particles in the SPHIR’s velocity range, this was identified as a 
secondary objective during the conceptual phase of the project. 

Design Requirements 

The output of the design phase has the largest impact on metrology, but is often the most 
overlooked. The operational specifications and measurement tolerances that come from 
the design phase can influence a project’s overall cost, the quality of the end product, or 
both. There are two very important elements, concerning metrology, which should be a 
part of every design. 

• Establish design tolerances that reflect the required functionality of the item. 

• Establish reasonable operational and/or processing requirements. 

For a measurement to be realistic (thus cost-effective), the specified tolerance must be 
integrally linked to a realistic functionality (or utility). In terms of metrology, this is the 
single most important consideration of the design phase. All designs have an optimum 
functional range, whether it is the voltage of a circuit or the diameter of a bolt hole. When 
a tolerance is set too far inside the functional range, the accept/reject criteria no longer 
reflects reality. Measurements are made to support the accept/reject decision. If a 
tolerance of ± 'A inch is adequate, specifying ± 0.010 inch does not add any value and most 
likely will increase the measurement cost. It will definitely increase the number of 
functional parts rejected. 

Conservative tolerances increase the probability that functional items will be rejected as 
out-of-tolerance. This is known as a false reject and can lead to unnecessary rework and 
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handling. Higher rejection rates also imply poorer production controls, which can create an 
excessively pessimistic view of the quality of the end-item production processes. This 
view may lead to more frequent disassembly and repair of end items than is necessary [11]. 

The following three examples illustrate the metrology issues arising from design tolerances 
and operational requirements that are not directly linked to the functional requirements 
which they support. 

The use of “conservative” tolerances (i.e., design tolerances which are intentionally 
restrictive) can lead to unintentional consequences, one of which is a false security that the 
process is achieving the design intent. This example, taken from a Space Shuttle 
Engineering Review Board (ERB) proceeding at KSC, underscores this point. 

The external airlock, which is used for Space Shuttle flight operations to the 
International Space Station (ISS), is attached to the Shuttle Payload Bay with latches 
that are bolted closed. The flight torque design requirement for this latch is 8000-8500 
inch-pounds for a maximum design load of 121,000 pounds. This prevents the latch 
from gapping at maximum loads. Typical flight loads for the airlock are less than 
14,000 pounds. 

A re-certification review of the custom-built equipment used to torque the latches led 
to an analysis of the overall torque process (equipment, people, and process). The 
results indicated the actual torque being applied to the latches fell between 6780 and 
10,900 inch-pounds. A Design Center evaluation of the latches determined a torque of 
6580 in-pounds was sufficient for flight loads less than 62,000 pounds. With the 
airlock flight loads being significantly less, the latch torque was acceptable for flight. 
While the torque equipment and process was being redesigned, per NASA policy, 
waivers to the design requirement were required from the Shuttle Program Office. 

This investigation revealed the lower requirement (8000 inch-pounds) was based on an 
analysis for maximum design limits of the latch, but the upper limit (8500 inch- 
pounds) was not linked to an analysis. It was also revealed in the ERB that a standard 
Class 3 torque (10,200 inch-pounds) was acceptable for the upper limit. Had the 
original design tolerances been less restrictive (i.e., closer to the functional 
requirements), then more solutions would have been available without an equipment 
redesign and flight waivers. 

Another possible consequence of “conservative” tolerances is a compounding of 
requirements that have the affect of “margin stacking.” Design tolerances that have 
additional margin built in to cover “unknown measurement errors” can directly conflict 
with Quality Assurance (QA) requirements which are in place to assure the measurements 
meet the design intent. 

Prior to Space Shuttle’s second post-Columbia return to flight mission (STS-121), 
NASA Safety and Mission Assurance (SMA) personnel became concerned Shuttle 
requirements for measurement assurance were not being followed [12]. The 
requirement stated that the measurement uncertainty shall be no more than ten percent 
of the required tolerance during inspections (known as the 10:1 rule). A short-term 
assessment was conducted to estimate the degree of non-compliance and identify any 
potential safety risks the non-compliance may have for the STS-121 mission. 
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The assessment only sampled the precision measurements made with various hand- 
held instruments on the STS-121 flight hardware. Approximately 67% did not meet 
the 10:1 requirement. 

Of the sampled measurements, 21% were found to be 2:1 or less which significantly 
increases the likelihood of the measurement being out of tolerance. These were sent to 
engineering for an evaluation of the possible impact to Shuttle safety. Engineering 
evaluations of these items determined that margins built into the Shuttle hardware 
would absorb the risk posed by the possible out-of-tolerance measurements. 

Operational and/or processing requirements should also be based on the unique product or 
system that it supports. Universal, or “one size fits all,” requirements are usually created 
with worst-case scenario criteria. In other cases, the costs of the measurement processes 
are higher without any additional benefits. 

At KSC, NASA’s Interface Control Document (ICD) required a very tight spacecraft 
weighing tolerance (±0.05% of actual weight) for all unmanned flights. Performance 
margins vary for different launches depending on the mission destination, number of 
stages, and even the time year. The spacecraft weight factors into the performance 
equation and depending on the amount of performance margin, knowing the precise 
weight can play a significant role in the success of the mission. 

In two recent unmanned missions, with similar size spacecraft, the performance 
margins were very different. For one launch, the performance margins were very 
limited, thus knowing precisely the weight of the spacecraft was critical to mission 
success (1285 ±0.64 kilograms per ICD). This required a very detailed weighing 
operation. 

The second spacecraft, scheduled to launch a few months later, had a much larger 
performance margin, thus the weight was not as critical. The second spacecraft’s 
weight was acceptable anywhere within 50 kilograms of the maximum weight 
allowance. Although not required for mission success, the ICD required the same level 
of precision in the weighing operation as the first spacecraft. 

Both launches were successful. Due to the analysis concerning weighing versus launch 
performance for these two launches, the ICD weighing requirement was changed to 
reflect mission specific performance requirements. 

Testing Metrology 

Testing, also known as experimentation, is about finding solutions to a problem. 
Metrology’s role in testing is a combination of research and design. The fundamental 
tenets of traceability and uncertainty are key features of testing, but valid results also 
require proper planning with well defined goals (i.e., what is the test trying to answer). 

The measurements, or data collection, are usually a small, albeit critical, portion of testing. 
An incomplete understanding of the data quality can lead to wrong conclusions about the 
testing results. As discussed earlier, uncertainty analysis provides an estimation of the data 
quality based on measurements. 
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This point is made explicitly clear in Coleman and Steele’s text on experimentation. 

Figure 2, taken from Coleman and Steele’s text, shows two model predictions with test 
data points. In Figure 2 a) it might be argued one model compares better with the data than 
the other. Once the uncertainty bands of the data are added in 2 b), any conclusions about 
either model validity becomes “fruitless” [13]. 




Figure 2: This figure is taken from Coleman and Steele’s Experimentation and Uncertainty Analysis for 
Engineers with the caption: “A comparison of mathematical model results with experimental data: (a) 
without uncertainty bands; (b) with uncertainty bands.” 

The results of the test must include the assumptions and the uncertainty; otherwise the 
limitations of the test may not be clear to reviewers. Again, this could lead to wrong 
conclusions. Wayne Hale, Space Shuttle Program Manager, in an email to NASA 
engineers titled, “Leading your leaders,” summarized reporting test results very succinctly. 

“Make sure that the test is well defined, and even then, it is important to explain to 
your leaders what the inherent accuracy (or error) of the test conditions or 
equipment have and what the assumptions or initial conditions were for the test. 
Test results without a good understanding of the accuracy of the test or the pedigree 
of the test assumptions are worth very little.” 
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Product Assurance Metrology 


Product assurance is at the end of the metrology chain and is very dependent on the input 
from each of the areas already discussed. As the old cliche says, “This is where the rubber 
meets the road.” 

At the simplest level, product assurance is about making sure end products and services 
meet the requirements as promised or specified. Traditionally, the emphasis of metrology 
in product assurance has been calibration of measuring instruments. Inspection processes 
utilize the calibrated instruments to verify the product specifications, where based on the 
measurement results, the decision is made to accept or reject and rework or scrap the item. 
Calibrated instruments provide traceability which is only one of the fundamental tenets. 
This alone does not assure the product meets the specified tolerances without the second 
tenet, uncertainty. 

As discussed earlier, measurement uncertainty analysis provides for the “goodness” of the 
measurement, but this is where problems arise in product assurance. Because 
measurement uncertainty analysis is usually beyond the scope and ability of the inspectors 
performing the work, it is rarely used for product assurance. Instead, many simplified 
methods, most developed over 50 years ago, are used. These simple methods were not 
designed to provide the type of evidence for product conformity required by AS9100 [14], 
which is a key quality requirement used by NASA and many aerospace companies. Proper 
inspection procedures must be developed which adequately accounts for the uncertainties 
in the inspection processes. 

Summary 

Data from measurements are fundamental elements in decision making. For modem 
consumers, Legal Metrology provides assurance that the measurement data going into their 
decisions are valid. For metrology to be transparent within NASA and other industries, it 
must be integrated throughout all phases of the life-cycle including research, design, 
testing and product assurance. 

There are many examples within NASA and industry, where better metrology could have 
saved money, time, and even lives. It takes very little thought for most program 
managers, project mangers, or system engineers (or anyone in the process) to remember an 
anecdotal story where better measurement data would have benefited their work. This 
includes those in research, design, testing, and end-item product assurance. 

The key to achieving an integrated metrology program is to standardize across the 
organization. All elements must be working from the same “playbook” to achieve the 
maximum benefits, which are fewer rejected items, higher quality, less costly inspection 
processes, etc. In other words, more confidence in data obtained from measurements. 


“ The more critical the decision, the more critical the data. The more critical the 
data, the more critical the measurement. ” 

NASA Reference Publication 1342 
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